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EXECUTIVE  SUMMARY 


Purpose 

The  purpose  of  this  paper  is  to  describe  models  and  guidelines  for  assessing  stan¬ 
dards  and  standards-based,  commercial  off-the-shelf  (COTS)  products  in  the  information 
technology  field.  The  models  are  intended  to  be  used  to  help  choose  among  competing  stan¬ 
dards  and  standards-based  COTS  products.  They  can  also  be  used  to  assess  a  single  stan¬ 
dard  or  product.  The  models  and  guidelines  can  be  applied  in  a  range  of  situations,  from  the 
selection  of  standards  and  products  for  a  single  specific  system  to  the  selection  of  recom¬ 
mended  standards  for  a  whole  class  of  systems,  as  is  done  in  the  DoD  Profile  of  Standards. 
There  is  a  corresponding  range  of  potential  users,  from  system  developers  to  program  man¬ 
agers  to  technical  architects. 

Background  and  Scope 

Standards,  according  to  the  International  Organization  for  Standardization,  are  doc¬ 
umented  agreements  containing  technical  specifications  or  other  precise  criteria  to  be  used 
consistently  as  rules,  guidelines,  or  definitions  of  characteristics,  to  ensure  that  materials, 
products,  processes  and  services  are  fit  for  their  purpose.  Standards-based  products  are 
products  that  adhere  to  the  technical  specifications  or  criteria  of  one  or  more  standards. 
Over  the  past  decade  and  more,  technology  has  driven  the  evolution  of  computer  systems 
from  large,  stand-alone,  proprietary  systems  based  on  a  single  powerful  processor  to  decen¬ 
tralized,  distributed  systems  composed  of  a  variety  of  processors  linked  together  via  com¬ 
munications  networks.  This  evolution  has  increased  customers  demands  that  computer 
system  components  from  different  vendors  be  able  to  communicate  and  exchange  informa¬ 
tion,  which  in  turn  has  led  to  an  increased  appreciation  of  the  value  of  standards.  Currently, 
there  are  a  large  number  of  existing  or  emerging  standards,  some  of  which  overlap  or 
address  the  same  technology  areas.  Users  of  standards  need  a  method  for  evaluating  and 
choosing  among  competing  standards  and  standards-based  products. 

The  standards  and  products  evaluation  models  described  herein  are  intended  to  pro¬ 
vide  the  user  with  specific  kinds  of  information  about  a  standard  or  product.  This  informa- 


tion  may  be  useful  in  making  certain  types  of  decisions,  specifically  decisions  regarding 
which  standards  or  products  of  multiple  ones  might  best  serve  identified  needs  or  decisions 
regarding  whether  a  given  standard  or  product  can  serve  those  needs. 

The  information  provided  by  these  models  cannot  be  applied  blindly  to  select  stan¬ 
dards  and  products.  They  must  be  used  in  conjunction  with  human  judgment  within  an 
overall  decision-making  process.  The  types  of  decisions  that  must  be  made  before  these 
models  are  applied  include  a  decision  as  to  where  standards  might  be  used  within  the  sys¬ 
tem  or  domain  along  with  an  understanding  of  what  benefits  and  costs  will  accme  as  a  result 
of  using  them  —  and  a  determination  of  which  standards  and  COTS  products  address  rel¬ 
evant,  needed  functionality.  To  use  the  results  of  the  standards  evaluation  model  or  products 
evaluation  model,  a  user  must  understand  the  needs  of  the  system  or  domain  under  consid¬ 
eration,  relative  to  the  evaluation  criteria  of  the  models,  and  relate  those  needs  to  the 
strengths  and  weaknesses  of  the  standards  and  products,  as  identified  by  the  evaluation 
models.  After  the  user  has  decided  to  adopt  a  particular  standard  or  use  a  particular  COTS 
product,  based  upon  the  evaluation  results,  the  user  should  identify  any  risks  associated 
with  that  choice  and  take  action  to  reduce  those  risks.  For  a  COTS  product,  such  actions 
would  include  a  viable  (structured)  demonstration  of  that  product  as  well  as  testing  that 
product  against  an  objective  set  of  tests. 

The  Evaluation  Models 

Standards  and  standards-based  products  are  best  chosen  by  considering  the  context 
in  which  they  will  be  used.  Relevant  context  includes  the  requirements  of  the  system  being 
acquired  or  developed,  its  architecture,  and  the  pool  of  relevant  standards  and  standards- 
based  products  fi'om  which  choices  can  be  made.  A  framework  is  presented  for  selecting 
standards  and  products  that  takes  into  account  the  context  in  which  they  will  be  used. 

This  document  presents  two  models,  one  for  evaluating  standards  and  one  for  eval¬ 
uating  standards-based  products.  Each  model  presents  a  set  of  criteria,  giving  a  brief 
description  of  each  criterion  and  why  that  criterion  is  important.  Related  criteria  are 
grouped  into  categories  so  that  a  standard  or  product  can  be  evaluated  by  category  instead 
of  by  individual  criteria. 

The  standards  evaluation  model  consists  of  15  criteria  grouped  into  four  categories, 
quality,  standards  support,  stability,  and  marketplace. 

•  Quality  —  The  six  criteria  of  completeness,  technical  quality,  unambiguous 
expression,  clarity,  strictly-defined  interface,  and  flexibility  address  the  quality 
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of  the  standard.  Poor  quality  standards  can  limit  the  degree  of  interoperability 
and  portability  of  products  based  on  them. 

•  Standards  support  —  The  support  for  a  standard  is  measured  by  the  credentials 
of  the  standard’s  approving  body,  by  the  standard’s  scope  of  acceptance,  and  by 
the  standard’s  conformance  specification. 

•  Stability  —  The  stability  of  domain,  lifecycle  of  standard,  and  stability  of  stan¬ 
dard  all  give  some  indication  of  how  mature  a  standard  is  and  how  long  it  is  like¬ 
ly  to  remain  influential.  A  stable  standard  is  desirable  because  change  in  a 
standard  can  be  expensive  to  those  who  have  adopted  it. 

•  Marketplace  —  Marketplace  considerations  can  determine  how  long  a  standard 
will  be  influential.  They  also  determine  the  amount  of  choice  there  will  be  in 
selecting  a  product  to  conform  to  the  standard.  There  are  three  criteria  in  the 
marketplace  category:  number  of  acceptable  products,  marketplace  presence, 
and  cost  of  standard. 

The  standards  products  evaluation  model  consists  of  21  criteria  grouped  into  seven 
categories:  quality,  integration  with  other  products,  standards  support,  product  support, 
manageability,  stability,  and  marketplace. 

•  Quality  —  The  quality  category  addresses  a  product’s  performance,  reliability, 
robustness,  functionality,  and  ease  of  use. 

•  Integration — Interoperability,  portability,  and  scalability  focus  on  the  ease  with 
which  a  product  can  be  integrated  with  other  products  and  into  other  environ¬ 
ments. 

•  Standards  support — The  two  criteria  in  this  category  address  the  product’s  con¬ 
formance  to  the  standard(s)  on  which  it  is  based:  conformance  accreditation  and 
degree  of  conformance.  They  are  applicable  only  to  products  that  are  based  on 
standards. 

•  Product  support  —  The  product  warranty  and  service  support  criteria  address 
how  much  support  a  purchaser  is  likely  to  get  from  a  vendor  if  problems  occur 
in  using  the  product. 

•  Manageability  —  This  deals  with  the  practical  concerns  in  bringing  a  product 
into  an  organization,  setting  it  up,  and  maintaining  it  for  use.  These  criteria  are: 
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ease  of  installation  and  integration,  training  and  professional  support,  license 
management,  and  remote  management. 

•  Stability  —  Vendor  stability  addresses  the  likelihood  that  the  vendor  will  remain 
in  business  to  support  the  product.  Product  maturity  addresses  the  robustness  of 
a  product  and  how  likely  and  how  often  it  needs  to  be  upgraded. 

•  Marketplace  —  Marketplace  considerations  can  determine  how  long  a  product 
will  be  supported  by  its  vendor  and  how  costly  it  will  be  to  use.  These  criteria 
are:  marketplace  presence,  third-party  support,  and  cost  of  product. 

The  models  are  flexible  and  can  be  tailored  to  a  given  situation.  Certain  criteria  can 
be  emphasized  or  de-emphasized,  according  to  situational  needs.  Further,  the  models  them¬ 
selves  can  easily  be  extended  or  modified  based  on  experience  in  their  use.  Criteria  can  be 
added  or  deleted  and  assessment  guidelines  further  developed.  As  the  models  are  more 
widely  used,  they  can  evolve  to  become  more  precise  instruments  for  evaluating  standards 
and  products. 
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L  INTRODUCTION 


1.1  PURPOSE 

The  purpose  of  this  paper  is  to  describe  models  and  guidelines  for  assessing  standards 
and  standards-based,  commercial  off-the-shelf  (COTS)  products  in  the  information  technology 
field.  The  models  are  intended  to  be  used  to  help  choose  among  competing  standards  and  stan¬ 
dards-based  COTS  products.  They  can  also  be  used  to  assess  a  single  standard  or  product.  The 
models  and  guidelines  can  be  applied  in  a  range  of  situations,  from  the  selection  of  standards 
and  products  for  a  single  specific  system  to  the  selection  of  recommended  standards  for  a  whole 
class  of  systems,  as  is  done  in  the  DoD  Profile  of  Standards  [DOD  93b].  There  is  a  correspond¬ 
ing  range  of  potential  users,  firom  system  developers  to  program  managers  to  technical  archi¬ 
tects. 


1.2  BACKGROUND 

Standards,  according  to  the  International  Organization  for  Standardization,  are  docu¬ 
mented  agreements  containing  technical  specifications  or  other  precise  criteria  to  be  used  con¬ 
sistently  as  rules,  guidelines,  or  definitions  of  characteristics,  to  ensure  that  materials,  products, 
processes  and  services  are  fit  for  their  purpose.  Standards-based  products  are  products  that 
adhere  to  the  technical  specifications  or  criteria  of  one  or  more  standards.  Over  the  past  decade 
and  more,  technology  has  driven  the  evolution  of  computer  systems  from  large,  stand-alone, 
proprietary  systems  based  on  a  single  powerful  processor  to  decentralized,  distributed  systems 
composed  of  a  variety  of  processors  linked  together  via  communications  networks.  This  evo¬ 
lution  has  increased  customers’  demands  that  computer  system  components  from  different  ven¬ 
dors  be  able  to  communicate  and  exchange  information,  which  in  turn  has  led  to  an  increased 
appreciation  of  the  value  of  standards.  Currently,  there  are  a  large  number  of  existing  or  emerg¬ 
ing  standards,  some  of  which  overlap  or  address  the  same  technology  areas.  Users  of  standards 
need  a  method  for  evaluating  and  choosing  among  competing  standards  and  standards-based 
products. 


1.3  SCOPE 


The  standards  and  products  evaluation  models  described  here  are  intended  to  provide 
the  user  with  specific  kinds  of  information  about  a  standard  or  product.  This  information  may 
be  useful  in  making  certain  types  of  decisions,  specifically  decisions  regarding  which  of  mul¬ 
tiple  standards  or  products  might  best  serve  identified  needs  or  decisions  regarding  whether  a 
given  standard  or  product  can  serve  those  needs. 

The  information  provided  by  these  models  cannot  be  applied  blindly  to  select  standards 
and  products.  They  must  be  used  in  conjunction  with  human  judgement,  within  an  overall  deci¬ 
sion-making  process.  The  types  of  decisions  that  must  be  made  before  these  models  are  applied 
include  a  decision  as  to  where  standards  might  be  used  within  the  system  or  domain  along  with 
an  understanding  of  what  benefits  and  costs  will  accrue  as  a  result  of  using  them;  and  a  deter¬ 
mination  of  which  standards  and  COTS  products  address  relevant,  needed  functionality.  In 
order  to  use  the  results  of  the  standards  evaluation  model  or  products  evaluation  model,  a  user 
must  understand  the  needs  of  the  system  or  domain  under  consideration,  relative  to  the  evalu¬ 
ation  criteria  of  the  models,  and  relate  those  needs  to  the  strengths  and  weaknesses  of  the  stan¬ 
dards  and  products,  as  identified  by  the  evaluation  models.  After  a  decision  has  been  made  to 
adopt  a  particular  standard  or  use  a  particular  COTS  product,  the  user  should  identify  any  risks 
associated  with  that  choice,  as  illuminated  by  the  evaluation  results,  and  take  actions  to  mitigate 
those  risks. 

1.4  APPROACH 

While  it  would  have  been  desirable  to  create  a  rigorously  predictive  quantitative  model 
for  evaluating  standards,  our  investigation  has  concluded  that  existing  knowledge  is  not  ade¬ 
quate  to  create  such  a  model.  Instead,  the  approach  taken  has  been  to  develop  a  more  subjective 
model  and  guidelines  that  provide  direction  in  understanding  how  to  evaluate  standards  and 
products,  a  checklist  to  insure  that  every  important  aspect  has  been  investigated,  and  a  codifi¬ 
cation  of  knowledge  about  the  desirability  of  different  aspects  of  standards  and  products. 

In  order  to  form  a  basis  for  the  model,  several  people,  both  within  and  outside  of  EDA, 
participated  in  discussions  concerning  the  characteristics  of  a  good  standard.  All  of  the  people 
interviewed  had  extensive  experience  in  the  standards  area,  many  having  participated  on  stan¬ 
dards  development  committees.  The  information  from  these  interviews  was  used  to  create  pre¬ 
liminary  sets  of  evaluation  criteria  for  standards  and  standards-based  products.  These  criteria 
were  refined  over  the  course  of  reviews  by  other  standards-knowledgeable  professionals.  The 
standards  evaluation  model  was  further  elaborated  by  creating  assessment  guidelines  for  each 
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of  its  criteria.  The  model  was  then  used  to  evaluate  ten  standards,  most  of  which  were  taken 
from  the  profile  of  standards  associated  with  the  DoD  Technical  Reference  Model.  As  a  result 
of  that  evaluation  exercise,  further  modifications  were  made  to  the  model. 

The  form  of  the  standards  evaluation  model  presented  here  is  very  similar  to  that  of  the 
model  developed  by  the  National  Institute  of  Standards  and  Technology  (NIST)  as  part  of  its 
Application  Portability  Profile  [NIST  95].  Both  models  contain  a  set  of  evaluation  criteria  and 
guidelines  for  assessing  a  standard  relative  to  the  criteria.  The  major  differences  are  in  the  num¬ 
ber  of  evaluation  criteria  (7  for  NIST,  15  for  the  standards  evaluation  model),  the  number  of 
assessment  levels  for  each  criterion  (3  for  NIST,  5  for  the  standards  evaluation  model),  and  the 
amount  of  guidance  given  to  the  user  in  assessing  a  standard. 

1.5  ORGANIZATION  OF  THE  DOCUMENT 

Standards  and  standards-based  products  are  best  chosen  by  considering  the  context  in 
which  they  will  be  used.  Relevant  context  includes  the  requirements  of  the  system  being 
acquired  or  developed,  its  architecture,  and  the  pool  of  relevant  standards  and  standards-based 
products  from  which  choices  can  be  made.  Chapter  2  presents  a  framework  for  selecting  stan¬ 
dards  and  products  that  takes  into  account  the  context  in  which  they  will  be  used. 

Two  models  are  presented  in  Chapter  3,  one  for  evaluating  standards  and  one  for  eval¬ 
uating  standards-based  products.  The  models  present  a  set  of  criteria  that  can  be  used  in  eval¬ 
uating  standards  and  products.  They  give  a  brief  description  of  each  criterion  that  may  include 
reasons  why  the  criterion  is  important.  The  models  are  meant  to  be  used  in  conjunction  with 
the  selection  framework  presented  in  Chapter  2. 

Chapter  4  develops  the  standards  evaluation  model  further  by  presenting  guidelines  for 
assessing  each  of  its  criteria.  It  also  offers  guidelines  for  using  and  tailoring  the  evaluation  mod¬ 
el.  Chapter  5  provides  a  summary  of  the  document. 


2.  SELECTION  FRAMEWORK 


Standards  and  standards-based  products  should  not  be  chosen  in  a  vacuum.  The 
requirements  and  architecture  of  the  system  being  built  are  important  sources  of  contextual 
information  that  should  guide  their  selection.  This  chapter  presents  a  four-tiered  framework, 
depicted  in  Figure  2-1,  that  can  be  used  to  guide  the  selection  of  standards  and  standards-based 
products. 


Figure  2-1.  Selection  Framework 


Ideally,  the  requirements  established  at  the  higher  tiers  should  determine  the  choices 
available  at  the  lower  tiers.  However,  as  the  double-headed  arrows  are  intended  to  indicate,  low¬ 
er  tiers  may  also  impact  choices  at  higher  tiers.  For  example,  the  availability  of  an  outstanding 
product  may  influence  the  choice  of  a  standard.  Even  the  highest  tier,  requirements,  is  not 
inunune  to  change  resulting  from  decisions  made  at  a  lower  level.  For  example,  in  many  large 
systems  such  as  the  Global  Command  and  Control  System  (GCCS)  and  the  Defense  Informa¬ 
tion  Infrastructure  (DO),  architecture  is  dictating  to  some  extent  which  requirements  will  be 
accommodated. 

Each  of  the  four  tiers.  Requirements,  Architecture,  Standards,  and  standards-based 
Products,  is  described  in  this  chapter.  The  final  section  of  the  chapter  discusses  a  process  for 
selecting  standards  and  products  using  the  selection  firamework,  the  evaluation  models  present¬ 
ed  in  Chapter  3,  and  the  assessment  guidelines  presented  in  Chapter  4.  The  selection  process 
can  be  used  in  different  contexts,  including  the  development  of  a  software  product,  the  creation 
of  a  software  system  composed  of  commercial  off-the-shelf  (COTS)  Juid  developed  products, 
and  the  development  of  guidance  (e.g.,  in  the  form  of  a  technical  architecture)  for  creating  soft¬ 
ware  systems. 

2.1  REQUIREMENTS 

The  top  tier  of  the  selection  framework  is  requirements.  Requirements  for  a  system 
include  functional  requirements,  non-functional  requirements,  and  external  requirements  or 
constraints.  Functional  requirements  describe  what  the  system  does  in  terms  of  the  inputs  and 
outputs  of  the  system  and  how  they  interrelate. 

Non-functional  requirements  delineate  non-functional  aspects  of  a  system  such  as  per¬ 
formance,  reliability,  portability,  cost,  etc.  Some  of  the  non-functional  requirements  such  as 
portability  may  express  the  needs  of  the  organization  acquiring  the  system,  while  others  such 
as-performance  more  directly  express  the  needs  of  the  user.  Non-functional  requirements  com¬ 
monly  address  areas  such  as  the  following: 

1.  Portability 

2.  Scalability 

3.  Interoperability 

4.  Flexibility 


6.  Performance 

7.  Robustness 

8.  Reliability 

9.  Availability 

10.  Security 

11.  Maintainability 

Many  non-functional  requirements  can  be  achieved  without  the  use  of  standards.  However, 
requirements  in  the  areas  of  portability  and  interoperability  typically  require  that  standards  be 
used. 

External  requirements  or  constraints  are  those  imposed  by  other  systems  or  by  a  parent 
organization.  A  common  example  is  the  requirement  to  be  compatible  with  existing  artifacts 
such  as  databases  or  applications  in  use  within  the  organization.  Other  examples  are  the  policy, 
legal,  and  environmental  requirements  established  by  the  DoD  and  imposed  on  all  organiza¬ 
tions  within  its  purview. 

2.2  ARCHITECTURE 

Architecture  is  defined  as  the  organizational  structure  of  a  system  or  component  [IEEE 
610.12].  Various  views  of  architecture  exist  in  the  open  systems  arena.  The  DoD  currently  is 
working  towards  a  definition  of  three  types  of  architecture:  operational,  systems,  and  technical. 
The  following  represent  DoD’s  proposed  definitions  of  the  three  types  of  architecture: 

1.  Operational  Architecture  covers  the  descriptions  of  the  tasks,  operational  ele¬ 
ments,  and  information  flows  required  to  accomplish  or  support  a  warfighting 
function. 

2.  Systems  Architecture  covers  the  descriptions,  including  graphics,  of  systems 
and  interconnections  providing  for  or  supporting  warfighting  functions. 

3.  Technical  Architecture  covers  a  minimal  set  of  rules  governing  the  arrangement, 
interaction,  and  interdependence  of  the  parts  or  elements  whose  purpose  is  to 
ensure  that  a  conformant  system  satisfies  a  specified  set  of  requirements. 

The  architecture  component  of  the  selection  framework  can  represent  different  ones  of 
these  architectures,  depending  on  the  context  in  which  the  framework  is  used.  For  example,  if 
used  in  the  context  of  a  systems  development,  it  represents  a  systems  architecture,  describing 


both  the  components  of  the  system  under  development  as  well  as  all  external  interfaces  of  the 
system  to  other,  existing  systems.  If  used  in  the  context  of  developing  guidance  for  a  whole 
domain  of  systems,  it  represents  a  technical  architecture. 

2.3  STANDARDS 

There  are  a  large  number  of  standards  in  existence  today.  It  is  estimated,  for  example, 
that  there  are  approximately  35,000  US  military  specifications  and  standards.  Even  in  the 
restricted  arena  of  information  technology,  there  are  hundreds  if  not  thousands  of  different  stan¬ 
dards  firom  which  to  choose. 

Many  standards  are  broad  in  coverage.  They  may  define  ranges  of  values  rather  than 
single  values  and  provide  options  for  adhering  to  the  standard.  Product  implementors  must 
decide  which  options  to  implement.  Users  must  decide  which  options  are  suited  to  their  needs. 

The  problem  is  compounded  when  multiple  broad  standards  must  be  chosen  to  operate 
together.  In  order  to  facilitate  the  selection  of  related,  complex  standards,  profiles  are  devel¬ 
oped.  A  profile  is  a  set  of  one  or  more  base  standards,  and,  where  applicable,  the  identification 
of  chosen  classes,  subsets,  options  and  parameters  of  those  base  standards,  necessary  for 
accomplishing  a  particular  function  PSO/IEC  95].  Profiles  constrain  a  user’s  choice  of  stan¬ 
dards  and  hence  aid  in  standards  selection.  One  of  the  most  visible  profiles  today  is  NIST’s 
Application  Portability  Profile  [NIST  95],  which  is  intended  to  direct  users  in  their  selection  of 
information  technology  standards  to  create  a  computing  environment  that  supports  portability. 
Another  widely-regarded  profile  is  the  DoD  Profile  of  Standards  [DOD  93b],  which  identifies 
standards  relevant  to  the  information  system  services  identified  in  the  Technical  Reference 
Model  (TRM)  of  the  TAFIM  [DOD  93a]. 

2.4  STANDARDS-BASED  PRODUCTS 

Some  standards  have  many  products  that  are  based  on  them,  while  other  standards  may 
have  none.  The  availability  of  products  that  conform  to  a  standard  can  be  an  important  consid¬ 
eration  in  the  selection  of  a  standard.  Likewise,  one  of  the  significant  issues  in  selecting  a  stan- 
dards-based  product  is  the  quality  of  the  standard  on  which  it  is  based.  The  selection  of  a 
standard  and  a  standards-based  product  are  often  interlinked. 

2.5  PROCESS  FOR  SELECTING  STANDARDS  AND  PRODUCTS 

Very  often,  standards  are  selected  implicitly  and  without  thought,  as  a  consequence  of 
the  choice  of  a  familiar  or  recommended  product.  While  circumstances  may  sometimes  dictate 
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that  decisions  be  made  in  a  bottom-up  order,  the  choices  made  at  every  level  of  the  selection 
framework  should  be  made  explicitly  and  after  consideration  of  the  alternatives  and  the  conse¬ 
quences. 

Ideally,  the  functional,  non-functional,  and  external  requirements  should  drive  every 
decision  that  is  made  in  implementing  the  system,  from  the  development  and  selection  of  an 
architecture  for  the  system  to  the  selection  of  standards  and  standards-based  products  that  will 
be  incorporated  into  the  system.  After  all,  without  the  necessity  for  satisfying  the  functional 
requirements,  there  would  be  no  point  in  developing  the  system  at  all.  Elaborating  the  ideal  sit¬ 
uation  furtheti  once  a  system  architecture  based  on  requirements  is  developed,  it,  along  with 
the  requirements,  should  drive  the  selection  of  standards;  and  once  standards  are  chosen,  they 
will  dictate  the  pool  of  products  from  which  a  final  choice  must  be  made. 

Often,  marketplace  and  other  realities  will  prescribe  a  different  order  of  decision  mak¬ 
ing.  It  might  be  that  the  choice  of  the  “best”  system  architecture,  based  solely  on  the  system 
requirements,  would  dictate  the  choice  of  an  immature  standard  or  of  a  standard  that  has  inad¬ 
equate  product  support,  while  the  choice  of  the  second  best  system  architecture  would  permit 
the  selection  of  a  mature  standard  supported  by  several  excellent  products. 

Therefore,  a  strict  top-down  decision-making  process,  where  system  architecture  is 
developed  based  solely  on  requirements,  standards  are  selected  based  on  system  architecture 
and  requirements,  and  products  are  chosen  always  after  standards  are  selected,  cannot  be  rec¬ 
ommended  in  all  situations.  What  is  recommended  instead  is  a  process  that  at  each  decision 
level  of  the  selection  framework  (system  architecture,  standards,  and  products)  looks  up  to 
ensure  that  all  requirements  from  the  levels  above  are  met  and  looks  down  to  assess  the  conse¬ 
quences  of  a  decision  on  the  levels  below.  Judgment  must  then  be  applied  in  making  the  deci¬ 
sion.  This  is  an  engineering  process  that  attempts  to  make  the  best  set  of  decisions  so  as  to 
maximize  the  benefits  on  the  eventual  developed  system  and  the  environment  in  which  it  is 
used. 

The  role  of  the  requirements  and  architecture  in  the  decision-making  process  for  select¬ 
ing  standards  is  summarized  in  Table  2-1.  The  need  for  using  standards  may  be  dictated  by  the 
non-functional  requirements  of  portability,  scalability,  and  interoperability,  which  typically  can 
only  be  satisfied  through  the  use  of  standards.  External  requirements  such  as  an  adopted  tech¬ 
nical  architecture  may  also  dictate  the  use  of  standards.  The  set  of  relevant  standards  will  be 
defined  by  the  functional  requirements,  external  requirements,  and  system  architecture,  and 
may  be  constrained  by  external  requirements.  Finally,  the  criteria  that  should  be  most  signifi¬ 
cant  in  assessing  a  standard  will  be  determined  by  non-functional  and  external  requirements. 


Table  2-1.  Roles  of  Requirements  and  Architecture  in  Selecting  Standards 


Requirements/Architecture 

Role 

Functional  Requirements 

Determine  set  of  functionaUy  relevant  standards 

Non-fiinctional  Requirements 

Create  need  for  standards 

Determine  most  relevant  standards  evaluation  criteria 

External  Requirements 

Create  need  for  standards 

Constrain  set  of  standards 

Determine  most  relevant  standards  evaluation  criteria 

System  Architecture 

Determine  set  of  architecturally  relevant  standards 

Once  a  set  of  relevant  standards  has  been  determined,  the  standards  evaluation  criteria 
(Chapter  3)  and  assessment  guidelines  (Chapter  4)  can  be  used  to  evaluate  the  competing  stan¬ 
dards.  The  results  of  the  evaluation  can  then  be  used  to  help  determine  which  standard  best 
meets  the  needs  of  the  organization,  as  well  as  to  identify  any  risks  associated  with  adopting 
that  standard. 

Often,  a  set  of  standards  must  be  chosen  with  the  requirement  that  each  individual  stan¬ 
dard  integrate  well  with  the  other  standards  in  the  set.  In  a  case  like  this,  the  selection  process 
can  be  broken  into  two  phases.  In  the  first  phase,  an  evaluation  is  done  of  how  well  individual 
standards  integrate  with  each  other.  In  the  second  phase,  evaluations  of  the  individual  standards 
can  be  done  using  the  standards  evaluation  model.  Information  from  both  phases  can  then  be 
used  to  choose  the  best  set  of  integrating  standards. 


3.  EVALUATION  MODELS 


This  chapter  presents  an  evaluation  model  for  standards  and  an  evaluation  model  for 
standards-based  products.  The  two  models  are  similar.  Each  consists  of  a  set  of  criteria  that  can 
be  used  to  assess  a  standard  or  a  standards-based  product.  Related  criteria  are  grouped  into  cat¬ 
egories,  so  that  a  standard  or  product  can  be  evaluated  by  category  instead  of  by  individual  cri¬ 
teria.  Table  3- 1  shows  the  standards  evaluation  model,  which  consists  of  fifteen  criteria  grouped 
into  four  categories,  and  the  products  evaluation  model,  which  consists  of  twenty-one  criteria 
grouped  into  seven  categories.  The  lists  of  criteria  have  been  developed  based  on  the  expertise 
of  several  professionals  knowledgeable  in  the  area  of  information  technology  standards.  How¬ 
ever,  the  lists  are  not  exhaustive,  and  other  experienced  professionals  may  have  a  different  opin¬ 
ion  as  to  which  evaluation  criteria  are  most  important. 

It  is  assumed  that  the  standards  and  products  being  evaluated  are  relevant  to  the  system 
or  domain  under  consideration.  There  is  no  criterion  that  evaluates  the  extent  to  which  a  stan¬ 
dard  or  product  addresses  the  needed  functionality.  By  leaving  a  functionality  criterion  out  of 
the  model,  it  has  in  effect  been  given  the  highest  possible  weight.  Standards  or  products  that 
are  not  functionally  relevant  are  not  considered  further,  no  matter  what  other  strengths  they  may 
possess. 

The  remainder  of  this  chapter  will  give  a  brief  description  of  each  of  the  evaluation  cri¬ 
teria  for  standards  and  standards-based  products.. 


Table  3-1.  Evaluation  Models  for  Standards  and  Standards-based  Products 


Evaluation 

Categories 

Standards  Evaluation  Criteria 

Products  Evaluation  Criteria 

Quality 

Completeness 

Technical  Quality 

Unambiguous  Expression 

Clarity 

Strictly-defined  Interface 

Flexibility 

Performance 

Reliability 

Robustness 

Functionality 

Ease  of  Use 

Integration 
with  Other 
Products 

Interoperability 

Portability 

Scalability 

Standards 

Support 

Qedentials  of  Approving  Body 
Scope  of  Acceptance 

Conformance  Specification 

Conformance  Accreditation 

Degree  of  Conformance 

Product 

Support 

Product  Warranty 

Service  Support 

Manage- 

abiUty 

Ease  of  Installation  &  Integration 
Training  and  Professional  Support 
License  Management 

Remote  Management 

Stability 

Stability  of  Domain 

Lifecycle  of  Standard 

Stability  of  Standard 

Vendor  Stability 

Product  Maturity 

Marketplace 

Number  of  Acceptable  Products 
Marketplace  Presence 

Cost  of  Standard 

Marketplace  Presence 

Third-party  Support 

Cost  of  Product 

3.1  EVALUATION  CRITERU  FOR  STANDARDS 

The  fifteen  evaluation  criteria  for  standards  are  grouped  into  four  categories:  quality, 
standards  support,  stability,  and  marketplace. 


3.1.1  Quality 

The  six  criteria  of  completeness,  technical  quality,  unambiguous  expression,  clarity, 
strictly-defined  interface,  and  flexibility  address  the  quality  of  the  standard.  Poor  quality  stan¬ 
dards  can  limit  the  degree  of  interoperability  and  portability  of  products  based  on  them. 

The  quality  criteria  are  not  independent  of  each  other.  Some  of  them  interrelate  to  pro¬ 
duce  desirable  results.  For  example,  the  criteria  of  completeness,  unambiguous  expression,  and 
strictly-defined  interface  work  together  to  produce  a  standard  whose  conforming  products  are 
more  likely  to  be  able  to  interoperate.  Others  of  the  quality  criteria  almost  oppose  each  other, 
so  that  it  may  not  be  possible  for  a  standard  to  score  highly  on  each  of  them.  For  example,  to 
achieve  flexibility  in  a  standard  may  require  that  areas  of  the  standard  deliberately  be  left 
incomplete,  so  as  to  allow  for  future  developments. 

Completeness 

A  standard  is  complete  if  it  addresses  all  functionality  that  is  relevant  and  necessary  to 
the  domain  and  scope  of  the  standard  and  if  each  area  of  functionality  is  fully  covered.  If  a  stan¬ 
dard  is  incomplete,  product  implementers  are  free  to  define  how  the  unaddressed  functionality 
will  be  provided.  This  will  have  a  negative  impact  on  interoperability. 

Technical  Quality 

Technical  quality  refers  to  the  quality  of  the  concept(s),  model(s),  and/or  technology 
that  are  central  to  the  standard.  It  addresses  such  issues  as  whether  they  are  simple  rather  than 
complex,  are  likely  to  yield  good  performance,  or  have  significant  limitations. 

Unambiguous  Expression 

A  standard  is  unambiguous  if  each  of  its  requirements  is  precisely  expressed  and  is  not 
open  to  interpretation.  Further,  the  requirements  as  a  group  must  be  self-consistent,  so  that  each 
can  be  conformed  to  without  impacting  conformance  to  any  other.  An  ambiguous  standard  is 
detrimental  to  interoperability. 

Clarity 

Clarity  refers  to  the  ease  with  which  the  written  standard  can  be  comprehended,  assum¬ 
ing  competence  in  any  technologies  underlying  the  standard. 
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Strictly-defined  Interface 

A  standard  has  a  strictly-defined  interface  if  it  does  not  provide  multiple  ways  for  an 
implementer  to  deliver  a  single  function,  and  does  not  permit  variations  in  the  interface  for  a 
single  function. 

Flexibility 

A  standard  is  flexible  if  it  makes  some  allowance  for  future  developments.  One  example 
of  a  flexibility  mechanism  is  a  self-defining  data  format  that  permits  data-interchange  standards 
to  handle  unanticipated  types  of  data.  Flexibility  mechanisms  help  keep  a  standard  from 
becoming  outdated  when  new  features  appear. 

3.1.2  Standards  Support 

The  support  for  a  standard  is  measured  by  the  credentials  of  the  standard’s  approving 
body,  by  the  standard’s  scope  of  acceptance,  and  by  the  standard’s  conformance  specification. 

Credentials  of  Approving  Body 

Credential  of  approving  body  encompasses  both  the  sphere  of  influence  of  the  bodies 
that  either  develop  or  endorse  the  standard,  and  their  expertise  in  developing  standards. 

Scope  of  Acceptance 

Scope  of  acceptance  addresses  the  breadth  of  the  community  that  uses  a  standard. 
Conformance  Specification 

Conformance  specification  deals  with  how  well  the  conformance  requirements  are 
defined  (e.g.,  specification  of  levels  of  conformance,  implementation  options,  test  suites)  and 
how  well  they  are  supported. 

3.1.3  Stability 

The  three  criteria  in  the  stability  category  (stability  of  domain,  lifecycle  of  standard,  and 
stability  of  standard)  give  some  indication  of  how  mature  a  standard  is  and  how  long  it  is  likely 
to  remain  influential.  A  stable  standard  is  desirable  because  change  in  a  standard  can  be  expen¬ 
sive  to  those  who  have  adopted  it. 
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Stability  of  Domain 

Stability  of  domain  measures  the  stability  of  the  hardware  and  software  technology  in 
the  domain  of  the  standard,  which  will  impact  the  stability  of  the  standard  itself.  A  stable 
domain  implies  that  a  standard  will  be  meaningful  for  a  longer  period  of  time. 

Lifecycle  of  Standard 

This  criterion  gives  some  indication  of  where  a  standard  is  in  its  lifecycle.  Is  it  still  in 
draft  form,  or  has  it  been  accepted  and  approved?  Is  it  still  in  its  prime  or  is  it  starting  to  be 
displaced  by  competing  standards? 

Stability  of  Standard 

The  stability  of  a  standard  refers  to  the  frequency  with  which  it  is  being  augmented  or 
changed;  when  the  expected  next  changes  are  to  occur,  the  pattern  of  changes;  and  whether  old¬ 
er  versions  of  the  standard  have  remained  compatible  with  newer  versions. 

3.1.4  Marketplace 

Marketplace  considerations  can  determine  how  long  a  standard  will  be  influential.  They 
also  determine  the  amount  of  choice  there  wiU  be  in  selecting  a  product  to  conform  to  the  stan¬ 
dard.  There  are  three  criteria  in  the  marketplace  category:  number  of  acceptable  products,  mar¬ 
ketplace  presence,  and  cost  of  standard. 

Number  of  Acceptable  Products 

This  criterion  indicates  the  amount  of  choice  available  in  selecting  a  product  to  support 
the  standard. 

Marketplace  Presence 

Marketplace  presence  refers  to  the  percentage  of  the  marketplace  covered  by  all  prod¬ 
ucts  adhering  to  the  standard,  relative  to  the  total  market  covered  by  all  products  of  all  compet¬ 
ing  standards. 

Cost  of  Standard 

Cost  of  standard  refers  to  the  cost  of  complying  with  the  standard  and  covers  whatever 
costs  are  perceived  by  the  user. 


3.2  EVALUATION  CRTTERU  FOR  STANDARDS-BASED  PRODUCTS 

The  twenty-one  evaluation  criteria  for  standards-based  COTS  products  have  been  orga¬ 
nized  into  seven  categories:  quality,  integration  with  other  products,  standards  support,  product 
support,  manageability,  stability,  and  marketplace.  Many  of  these  evaluation  criteria  are  appli¬ 
cable  to  products  that  have  not  been  based  on  standends  as  well  as  to  those  that  have  been  based 
on  standards. 

3.2.1  Quality 

The  quality  category  addresses  a  product’s  performance,  reliability,  robustness,  func¬ 
tionality,  and  ease  of  use. 

Performance 

Most  products  have  performance  requirements  placed  upon  them.  Performance  require¬ 
ments  usually  address  speed,  but  may  also  address  throughput,  capacity,  etc.  There  are  several 
ways  that  speed  requirements  can  be  expressed,  e.g.,  average  speed,  worst-case  speed,  etc. 
Depending  on  the  situation,  a  product  may  have  to  be  evaluated  for  several  different  perfor¬ 
mance  criteria. 

Reliability 

The  reliability  of  a  product  refers  to  the  dependability  with  which  it  operates  correctly. 
Reliability  is  often  measured  as  the  mean  time  between  failures. 

Robustness 

Robustness  measures  a  product’s  response  to  unanticipated  or  incorrect  inputs.  Can  it 
continue  operating?  Does  it  shut  down  gracefully,  without  losing  critical  data? 

Functionality 

The  functionality  of  a  product  is  measured  by  the  extent  to  which  it  satisfies  identified 
user  needs.  This  criterion  is  related  to  the  product  maturity  criterion,  in  that  it  is  likely  that  more 
mature  products  will  have  more  functionality. 

Ease  of  Use 

This  category  measures  how  easy  a  product  is  to  use,  whether  by  a  novice  or  experi¬ 
enced  user. 
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3.2.2  Integration  with  Other  Products  and  with  Environment 

The  criteria  under  the  integration  category  focus  on  the  ease  with  which  a  product  can 
be  integrated  with  other  products  and  into  other  environments. 

Interoperability 

Interoperability  is  defined  as  the  ability  of  two  or  more  systems  or  components  to 
exchange  and  use  information  [IEEE  6 10. 12].  Interoperability,  thus,  is  not  a  property  of  a  single 
product,  but  rather  a  property  of  two  or  more  products  relative  to  each  other.  To  the  extent  that 
a  product  uses  standard  communication  protocols,  data  interchange  formats,  and  distributed 
system  interfaces  to  send,  receive,  and  use  information,  it  will  be  more  likely  to  interoperate 
with  other  products. 

Portability 

Portability  is  the  ease  with  which  a  system  or  component  can  be  transferred  firom  one 
hardware  or  software  environment  to  another  [IEEE  610.12].  The  portability  of  a  product  can 
be  enhanced  by  developing  it  using  a  standardized  programming  language  and  standardized 
interfaces  to  its  computing  environment. 

Scalability 

Scalability  refers  to  the  ability  to  use  application  software  on  many  different  classes  of 
hardware/software  platforms  firom  personal  computers  to  super  computers. 

3.2.3  Standards  Support 

The  two  criteria  in  this  category  address  the  product’s  conformance  to  the  standard(s) 
on  which  it  is  based.  They  are  applicable  only  to  products  that  are  based  on  standards. 

Conformance  Accreditation 

This  criterion  evaluates  the  strength  of  any  claims  made  concerning  the  product’s  con¬ 
formance  to  standards.  For  example,  claims  made  by  the  vendor  but  not  supported  by  any  test¬ 
ing  would  be  considered  weak  claims.  At  the  other  extreme,  conformance  claims  based  on  the 
execution  of  an  accepted  conformance  test  suite  by  a  third-party  testing  organization  (i.e.,  an 
organization  independent  of  the  vendor  and  the  purchaser)  would  be  considered  very  strong. 
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Degree  of  Conformance 

This  criterion  measures  how  much  of  a  standard  the  product  conforms  to,  and  to  what 
depth  it  conforms.  It  also  addresses  a  product’s  known  extensions  to  a  standard,  since  they  may 
affect  a  product’s  portability. 

3.2.4  Product  Support 

The  criteria  in  this  category  address  how  much  support  a  purchaser  is  likely  to  get  from 
a  vendor  if  problems  occur  in  using  the  product. 

Product  Warranty 

Hardware  products  often  come  with  a  warranty  that  the  product  will  work  correctly  for 
some  period  of  time.  Software  will  assuredly  come  with  a  trivial  warranty  that  protects  the  pur¬ 
chaser  if  the  software  has  been  corrupted  on  the  distribution  media.  However,  stronger  warran¬ 
ties  should  be  available  to  assure  that  the  software  will  operate  correctly  in  the  purchaser’s 
environment. 

Service  Support 

This  criterion  measures  the  extent  to  which  the  user  is  supported  if  he  or  she  has  diffi¬ 
culties  with  the  product.  Is  there  phone-line  support,  on-site  support,  bulletin-board  support?  Is 
a  collection  of  common  problems  and  solutions  available  for  customers  to  peruse?  Are  bug  fix¬ 
es  conveniently  available?  How  far  does  the  vendor  go  to  make  sure  that  customers  are  satis¬ 
fied? 

3.2.5  Manageability 

The  manageability  criteria  deal  with  the  practical  concerns  in  bringing  a  product  into 
an  organization,  setting  it  up,  and  maintaining  it  for  use. 

Ease  of  Installation  and  Integration 

This  criterion  measure  the  ease  with  which  a  product  can  be  installed  and  then  integrat¬ 
ed  into  an  existing  hardware/software  environment. 

Training  and  Professional  Support 

This  criterion  measures  the  amount  of  training  needed  for  end  users,  support  personnel, 
and  system  administrators  to  be  able  to  deal  with  and  use  the  product,  the  availability  of  train¬ 
ing,  and  the  amount  of  on-going  professional  support  required  for  optimum  use  of  the  product. 
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For  some  products,  the  end  user  needs  no  support;  for  others,  perhaps  only  occasional  support 
is  needed;  while  for  still  others  a  fairly  constant  access  to  technical  support  staff  may  be 
required. 

License  Management 

License  management  measures  how  difficult  it  is  to  manage  the  licensing  for  the  prod¬ 
uct,  whether  there  be  one  license  for  each  user,  a  token-based  licensing  scheme,  or  an  unlimited 
group  license. 

Remote  Management 

This  criterion  measures  the  ease  with  which  a  product  can  be  managed  from  a  central 
site.  It  is  most  important  for  large  organizations  that  want  to  centralize  the  management  of 
products  used  throughout  the  organization. 

Training  Requirements 

This  criterion  measures  the  amount  of  training  needed  and  available  for  end  users,  sup¬ 
port  personnel,  and  system  administrators  to  be  able  to  deal  with  and  use  the  product. 

3.2.6  Stability 

The  stability  criteria  deal  with  the  maturity  of  a  product  and  the  stability  of  its  vendor. 
Vendor  Stability 

Vendor  stability  measures  the  likelihood  that  the  vendor  will  remain  in  business.  This 
is  important  if  upgrades  to  the  product  are  desired  or  if  support  for  the  product  is  needed  for  a 
significant  period  of  time. 

Product  Maturity 

Product  maturity  is  a  measure  of  how  likely  a  product  is  to  be  upgraded  with  new  fea¬ 
tures.  It  is  also  a  measure  of  how  error-free  the  product  is.  Maturity  is  often  but  not  always  relat¬ 
ed  to  the  age  of  a  product,  because  over  time  more  features  are  typically  added  and  hopefully, 
more  problems  are  removed. 


3.2.7  Marketplace 

Marketplace  considerations  can  determine  how  long  a  product  will  be  supported  by  its 
vendor  and  how  costly  it  will  be  to  use.  There  are  three  criteria  in  the  markeq)lace  category: 
marketplace  presence,  third-party  support,  and  cost  of  product 

Marketplace  Presence 

This  criterion  measures  the  percentage  of  the  market  captured  by  the  product  relative 
to  the  total  market  shared  by  all  competing  products  that  support  the  same  standard(s).  A  prod¬ 
uct  with  more  market  share  is  likely  to  be  supported  for  a  longer  period  of  time. 

Third-party  Support 

This  criterion  measures  the  amount  of  third-party  support  available  for  a  product, 
including  training,  technical  support,  and  value-added  capabilities. 

Cost  of  Product 

Cost  of  product  refers  to  whatever  costs  are  perceived  by  the  user  as  part  of  the  cost  of 
using  the  product  It  might  include  items  such  as  the  initial  purchase  price,  upgrade  costs,  and 
support  costs,  as  well  as  costs  associated  with  training  and  technical  support  for  the  product 


4.  GUroELINES  FOR  ASSESSING  A  STANDARD  RELATIVE  TO  THE 

EVALUATION  CRITERIA 


# 


The  first  part  of  this  chapter  will  give  guidelines  for  assessing  a  standard  relative  to  the 
evaluation  criteria  in  the  standards  evaluation  model.  Assessments  will  be  made  on  a  five-point 
scale,  with  1  representing  the  lowest  score  and  5  the  highest.  These  rating  levels  are  meant  to 
correspond  to  a  grading  of  poor,  fair,  average,  good,  and  excellent,  respectively.  The  guidelines 
for  level  5  will  be  presented  first,  then  those  for  level  4,  and  so  forth.  The  first  level  that  accu¬ 
rately  describes  the  standard  represents  the  assessment  of  the  standard  relative  to  that  criterion. 
Some  criteria  may  not  use  all  five  assessment  levels.  For  example,  the  flexibility  criterion  uses 
only  three  of  the  five  assessment  levels,  excellent,  average,  and  poor. 

The  assessment  guidelines  for  the  standards  evaluation  criteria  are  phrased  using  objec¬ 
tive  terms  wherever  possible.  Some  guidelines  however  use  subjective  terms  and  require  judg¬ 
ment  to  be  applied.  Further,  the  judgment  can  be  based  on  a  great  deal  or  very  little  research 
into  the  standard  and  technologies  in  question.  The  guidelines  cannot  make  the  evaluation  a 
rote  activity.  What  they  can  do  is  guide  the  reviewer  toward  making  assessments  in  a  consistent 
way,  considering  the  same  qualifications  for  each  standard  that  is  evaluated. 

As  much  as  possible,  the  assessment  guidelines  are  stated  in  such  a  way  as  to  allow  a 
standard  to  be  assessed  in  absolute  terms  rather  than  assessed  relative  to  another  standard  or 
relative  to  a  particular  situation.  The  major  advantage  of  this  approach  is  that  a  single  evaluation 
of  a  standard  can  be  used  by  many  different  organizations  and  can  be  compared  against  the 
evaluation  of  any  other  related  standard.  The  one  criterion  that  has  been  expressed  in  relative 
terms  is  the  cost  criterion. 

The  evaluation  model  and  assessment  guidelines  can  be  used  to  compare  two  or  more 
standards.  They  can  also  be  used  to  compare  a  single  standard  against  the  requirements  of  a 
given  organization  for  a  given  system.  Both  the  standard  and  the  organizational  needs  could  be 
assessed  relative  to  the  standards  evaluation  model.  A  judgement  could  then  be  made  of  how 
well  the  standard  meets  the  organizational  needs. 
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Assessments  can  be  compared  at  the  level  of  the  fifteen  individual  criteria.  Alternative¬ 
ly,  evaluations  of  the  criteria  within  categories  can  be  combined  for  overall  category  assess¬ 
ments  which  can  then  be  compared.  The  results  of  the  assessments  can  be  displayed  in  a  tabular 
format,  in  a  line  graph  or  Kiviat  diagram,  in  bar  charts,  or  in  any  other  desired  way. 

While  several  additional  significant  criteria  could  be  added  to  the  standards  evaluation 
model,  it  is  likely  that  the  present  fifteen  criteria  may  be  too  numerous  to  allow  for  a  practical 
and  focused  evaluation.  Section  4.2  gives  some  guidelines  for  tailoring  the  use  of  the  evaluation 
model  to  reflect  the  most  important  needs  of  the  organization.  Section  4.3  illustrates  the  use  of 
the  assessment  guidelines  with  an  example  standard  evaluation. 

4.1  ASSESSMENT  GUIDELINES 

4.1.1  Quality 
Completeness 

A  standard  is  complete  if  it  addresses  all  functionality  that  is  relevant  and  necessary  to  the 
domain  and  scope  of  the  standard  and  if  each  area  of  functionality  is  fully  covered.  If  a  stan¬ 
dard  is  incomplete,  product  implementers  are  free  to  define  how  the  unaddressed  functionality 
will  be  provided.  This  will  have  a  negative  impact  on  interoperability. 

In  the  guidelines  for  evaluating  completeness,  the  word  ‘major’  refers  to  functionality  that  is 
central  to  the  domain  of  the  standard,  while  ‘minor’  refers  to  functionality  that,  while  impor¬ 
tant,  is  more  on  the  periphery  and  may  not  be  needed  in  all  circumstances.  A  particular  area  of 
functionality  is  fully  covered  if  every  significant  aspect  of  it  is  addressed. 

5.  All  relevant  areas  of  functionality,  major  and  minor,  are  addressed  and  fully  cov¬ 
ered. 

4.  Only  one  minor  area  of  functionality  is  not  addressed  or  is  not  fully  covered.  All 
major  areas  of  functionality  are  addressed  and  fully  covered. 

3.  At  most  two  major  areas  and  two  minor  areas  of  functionality  are  not  addressed  or 
are  not  fuUy  covered. 

2.  At  most  two  major  areas  and  two  minor  areas  of  functionality  are  not  addressed. 

1.  More  than  two  major  areas  of  functionality  are  not  addressed. 
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Technical  Quality 

Technical  quality  refers  to  the  quality  of  the  concept(s),  model(s),  and/or  technology  that  are 
central  to  the  standard.  It  addresses  such  issues  as  whether  they  are  simple  rather  than  com¬ 
plex,  are  likely  to  yield  good  performance,  or  have  significant  limitations. 

5.  Central  concepts,  models,  and/or  technology  of  the  standard  are  well-suited  to  their 
purpose  and  are  well-understood  (e.g.,  reference  models  or  formal  models  are  avail¬ 
able).  They  are  elegant  and  simple,  provide  excellent  performance  in  areas  relevant 
to  the  domain  (e.g.,  speed,  reliability),  and  have  no  known  limitations  relevant  to  the 
domain  of  the  standard. 

4.  Central  concepts,  models,  and/or  technology  of  the  standard  are  well-suited  to  their 
purpose  and  are  well-understood.  They  have  few  areas  of  complexity,  provide  good 
performance  in  areas  relevant  to  the  domain  (e.g.,  speed,  reliability),  and  have  at 
most  a  few  minor  limitations  relevant  to  the  domain  of  the  standard. 

3.  Central  concepts,  models,  and/or  technology  of  the  standard  are  suited  to  their  pur¬ 
pose.  They  are  more  simple  than  complex,  provide  acceptable  performance  in  areas 
relevant  to  the  domain  (e.g.,  speed,  reliability),  and  have  minor  limitations  and  /or 
at  most  one  major  limitation  relevant  to  the  domain  of  the  standard. 

2.  Central  concepts,  models,  and/or  technology  of  the  standard  are  somewhat  suited  to 
their  purpose.  They  may  not  be  completely  understood  or  proven.  They  have  signif¬ 
icant  areas  of  complexity,  provide  minimally  acceptable  performance  in  areas  rele¬ 
vant  to  the  domain  (e.g.,  speed,  reliability),  and  have  two  or  more  major  limitations 
relevant  to  the  domain  of  the  standard. 

1.  Central  concepts,  models,  and/or  technology  of  the  standard  are  not  well-suited  to 
their  purpose.  They  have  significant  areas  of  complexity,  provide  unacceptable  per¬ 
formance  in  at  least  one  area  relevant  to  the  domain  (e.g.,  speed,  reliability),  and 
have  two  or  more  major  limitations  relevant  to  the  domain  of  the  standard. 

Unambiguous  Expression 

A  standard  is  unambiguous  if  each  of  its  requirements  is  precisely  expressed  and  is  not  open 
to  interpretation.  Further,  the  requirements  as  a  group  must  be  self-consistent,  so  that  each  can 
be  conformed  to  without  impacting  conformance  to  any  other.  An  ambiguous  standard  is  det¬ 
rimental  to  interoperability. 


5.  The  standard  states  all  requirements  explicitly,  using  precise  language  and  provid¬ 
ing  objective  criteria  which  must  be  satisfied  in  order  to  conform  to  the  standard. 
Each  requirement  is  consistent  with  all  of  the  others. 

4.  The  standard  states  all  requirements  explicitly,  but  in  a  few  minor  cases  language 
may  be  imprecise  or  objective  criteria  for  satisfying  the  standard  may  be  missing. 
Each  requirement  is  consistent  with  all  of  the  others. 

3.  A  few  minor  requirements  are  implicit  or  are  stated  using  imprecise  language  and 
giving  either  subjective  criteria  or  no  criteria  for  satisfying  the  standard.  Each 
requirement  is  consistent  with  all  of  the  others. 

2.  One  or  two  significant  requirements  are  implicit  or  are  stated  using  imprecise  lan¬ 
guage  and  giving  either  subjective  criteria  or  no  criteria  for  satisfying  the  standard. 
There  are  at  most  two  inconsistencies  among  the  requirements  of  the  standard. 

1 .  Several  significant  requirements  are  implicit  or  are  stated  using  imprecise  language 
and  giving  either  subjective  criteria  or  no  criteria  for  satisf3dng  the  standard.  There 
may  be  more  than  two  inconsistencies  among  the  requirements  of  the  standard. 

Clarity 

Clarity  refers  to  the  ease  with  which  the  written  standard  can  be  comprehended,  assuming 
competence  in  any  technologies  underlying  the  standard. 

5.  It  is  easy  to  read  and  understand  the  individual  sentences  of  the  standard,  and  easy 
to  understand  all  major  and  minor  requirements  of  the  standard.  A  description  of  the 
rationale  for  the  standard  and  the  contexts  in  which  it  is  intended  to  be  used,  as  well 
as  the  definition  of  significant  terms  are  included  as  part  of  the  standard. 

4.  It  is  easy  to  read  and  understand  the  individual  sentences  of  the  standard,  and  easy 
to  gain  an  understanding  of  all  major  and  minor  requirements  of  the  standard. 

3.  It  is  difficult  to  read  and  understand  some  of  the  individual  sentences  of  the  stan¬ 
dard,  and/or  requires  some  effort  to  gain  an  understanding  of  the  major  and  minor 
requirements  of  the  standard. 

2.  It  is  difficult  to  read  and  understand  some  of  the  individual  sentences  of  the  stan¬ 
dard,  and/or  requires  significant  effort  to  gain  an  understanding  of  the  major 
requirements  of  the  standard. 
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1.  It  is  difficult  to  read  and  understand  many  of  the  individual  sentences  of  the  stan¬ 
dard,  and/or  requires  significant  effort  to  gain  even  a  partial  understanding  of  the 
major  requirements  of  the  standard. 

Strictly-defined  Interface 

A  standard  has  a  strictly-defined  interface  if  it  does  not  provide  multiple  ways  for  an 
implementor  to  deliver  a  single  function,  and  does  not  permit  variations  in  the  interface  for  a 
single  function. 

5.  All  aspects  of  the  interface  are  specifically  and  concretely  defined,  giving  no 
options  to  the  implementor. 

4.  One  or  two  functions  in  the  interface  allow  a  range  of  permissible  parameter  values 
rather  than  a  single  permissible  value.  All  other  aspects  of  the  interface  are  specifi¬ 
cally  and  concretely  defined,  giving  no  options  to  the  implementor. 

3.  Several  functions  in  the  interface  allow  a  range  of  permissible  parameter  values 
rather  than  a  single  permissible  value.  All  other  aspects  of  the  interface  are  specifi¬ 
cally  and  concretely  defined,  giving  no  options  to  the  implemented 

2.  One  or  two  functions  in  the  interface  can  be  provided  in  more  than  one  way.  All  oth¬ 
er  aspects  of  the  interface  are  specifically  and  concretely  defined,  giving  no  options 
to  the  implemented 

1 .  Several  functions  in  the  interface  can  be  provided  in  more  than  one  way. 
Flexibility 

A  standard  is  flexible  if  it  makes  some  allowance  for  future  developments.  One  example  of  a 
flexibility  mechanism  is  a  self-defining  data  format  that  permits  data-interchange  standards  to 
handle  unanticipated  types  of  data.  Flexibility  mechanisms  help  keep  a  standard  fi'om  becom¬ 
ing  outdated  when  new  features  appear. 

5.  Multiple  flexibility  mechanisms,  allowing  for  future  and  unforeseen  developments, 
have  been  defined  as  part  of  the  interface  of  the  standard. 

3.  A  single  flexibility  mechanism,  allowing  for  future  and  unforeseen  developments, 
has  been  defined  as  part  of  the  interface  of  the  standard. 

1.  No  flexibility  mechanism,  allowing  for  future  and  unforeseen  developments,  has 
been  defined  as  part  of  the  interface  of  the  standard. 
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A,\2  Support 


Credentials  of  Approving  Body 

Credential  of  approving  body  encompasses  both  the  sphere  of  influence  of  the  bodies  that 
either  develop  or  endorse  the  standard,  and  their  expertise  in  developing  standards.  The  term 
“standard”  is  used  loosely  to  include  specifications  that  have  been  produced  outside  of  a  stan¬ 
dards  organization,  as  well  as  specifications  that  are  represented  by  products  rather  than  docu¬ 
ments. 

5.  The  standard  has  been  developed  or  endorsed  by  an  international  standards  organi¬ 
zation  or  U.S.  national  standards  organization. 

4.  The  standard  has  been  developed  or  endorsed  by  a  consortium,  forum,  or  special 
interest  group. 

3.  The  standard  has  been  developed  or  endorsed  by  the  U.S.  Department  of  Defense. 

2.  The  standard  is  a  de  facto  or  company-proprietary  standard. 

1.  All  other  standards,  including  those  developed  by  a  local  organization  for  internal 
use. 

Scope  of  Acceptance 

Scope  of  acceptance  addresses  the  breadth  of  the  community  that  uses  a  standard.  The  term 
“standard”  is  used  loosely  to  include  specifications  that  have  been  produced  outside  of  a  stan¬ 
dards  organization,  as  well  as  specifications  that  are  represented  by  products  rather  than  docu¬ 
ments. 

5.  The  standard  is  accepted  by  a  community  that  is  international  and  that  spans  the 
domain(s)  in  which  the  standard  is  applicable. 

4.  The  standard  is  accepted  by  a  U.S.  national  community  that  spans  the  domain(s)  in 
which  the  standard  is  applicable. 

3.  The  standard  is  accepted  internationally  or  throughout  the  U.S.  by  a  community  that 
spans  a  significant  segment  of  the  domain(s)  in  which  it  is  applicable. 

2.  The  standard  is  accepted  regionally  by  a  community  that  spans  the  domain(s)  in 
which  the  standard  is  applicable. 

1.  The  standard  is  accepted  in  a  smaller  geographic  region  or  by  a  community  that 
spans  a  small  segment  of  the  domain(s)  in  which  the  standard  is  applicable. 


Conformance  Specification 

Conformance  specification  deals  with  how  well  the  conformance  requirements  are  defined 
(e.g.,  specification  of  levels  of  conformance,  implementation  options,  test  suites)  and  how 
well  they  are  supported. 

5.  All  seven  of  the  following  characteristics  are  true:  test  suites  have  been  defined  as 
part  of  the  standard;  the  defined  test  suites  account  for  permitted  levels  of  conform¬ 
ance  and  for  implementation  options;  executable  test  suites,  conforming  to  those 
defined  in  the  standard,  exist;  executable  test  suites  demonstrating  multi-vendor 
interoperability  exist;  testing  organizations  are  available  to  execute  available  test 
suites;  testing  organizations  are  certified  and  independent  of  vendors  and  users;  a 
reference  implementation  for  the  standard  exists  and  is  readily  available. 

4.  Five  or  six  of  the  seven  characteristics  listed  above  are  true. 

3.  Three  or  four  of  the  seven  characteristics  listed  above  are  true. 

2.  One  or  two  of  the  seven  characteristics  listed  above  are  true. 

1 .  None  of  the  seven  characteristics  listed  above  are  true. 

4.1.3  Stability 
Stability  of  Domain 

Stability  of  domain  measures  the  stability  of  the  hardware  and  software  technology  in  the 
domain  of  the  standard,  which  will  impact  the  stability  of  the  standard  itself.  A  stable  domain 
implies  that  a  standard  will  be  meaningful  for  a  longer  period  of  time.  More  impact  is  attrib¬ 
uted  to  a  change  in  hardware  technology  than  to  one  in  software  technology. 

5.  No  technology  in  the  domain  appears  to  be  changing. 

4.  Incidental  software  technology  in  the  domain  is  changing. 

3.  Incidental  hardware  technology  in  the  domain  is  changing. 

2.  Significant  software  technology  in  the  domain  is  changing. 

1 .  Significant  hardware  technology  in  the  domain  is  changing 
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Lifecycle  of  Standard 

This  criterion  gives  some  indication  of  where  a  standard  is  in  its  lifecycle.  Is  it  still  in  draft 
form,  or  has  it  been  accepted  and  approved?  Is  it  still  in  its  prime  or  is  it  starting  to  be  dis¬ 
placed  by  competing  standards? 

In  the  case  of  a  “standard”  that  is  represented  only  by  produces),  “approval”  should  be  inter¬ 
preted  as  acceptance  by  users  (user  base). 

5.  The  standard  has  been  approved.  Its  acceptance  appears  to  be  increasing  or  remain¬ 
ing  steady. 

4.  The  standard  has  been  approved.  Its  acceptance  appears  to  be  declining  at  a  slow 
rate,  as  competing  standards  start  to  appear  in  draft  form. 

3.  The  standard  is  in  mature  draft  form  and  appears  to  be  near  approval.  Its  acceptance 
appears  to  be  increasing. 

2.  The  standard  is  in  early  draft  form.  Its  acceptance  is  light 

1.  The  standard  is  obsolete  or  near  obsolete.  Its  underlying  technology  is  being 
replaced.  Competing  standards  are  mature. 

Stability  of  Standard 

The  stability  of  a  standard  refers  to  the  frequency  with  which  it  is  being  augmented  or 
changed;  when  the  expected  next  changes  are  to  occur;  the  pattern  of  changes;  and  whether 
older  versions  of  the  standard  have  remained  compatible  with  newer  versions. 

5.  The  standard  is  stable.  No  updates  have  occurred  for  over  five  years  and  none  are 
currently  planned. 

4.  The  standard  experiences  predictable  updates  occurring  with  average  frequency  of 
once  every  five  years  or  less.  Updates  represent  additions  to  the  standard  rather  than 
changes  to  existing  functionality,  so  that  products  conforming  to  older  versions  of 
the  standard  will  still  conform  to  (a  subset  of)  the  newer  version. 

3.  The  standard  experiences  predictable  updates  occurring  with  average  frequency  of 
once  every  three  years  or  less.  Updates  represent  additions  to  the  standard  rather 
than  changes  to  existing  functionality,  so  that  products  conforming  to  older  versions 
of  the  standard  will  still  conform  to  (a  subset  of)  the  newer  version. 
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2.  The  standard  experiences  predictable  updates  occurring  with  average  frequency  of 
once  every  year  or  less.  Updates  sometimes  represent  changes  to  the  standard,  so 
that  products  conforming  to  older  versions  of  the  standard  do  not  conform  to  the 
newer  version  of  the  standard. 

1.  The  standard  is  new  enough  that  its  stability  cannot  be  determined  or,  the  standard 
experiences  unpredictable  updates  occurring  at  irregular  intervals.  Updates  repre¬ 
sent  changes  to  the  standard  so  that  products  conforming  to  older  versions  of  the 
standard  do  not  conform  to  the  newer  version  of  the  standard. 

4.1.4  Marketplace 

Number  of  Acceptable  Products 

This  criterion  indicates  the  amount  of  choice  available  in  selecting  a  COTS  product  to  support 
the  standard. 

5.  A  large  number  (6  or  more)  of  commercial  products,  spanning  the  range  from  min¬ 
imal  functionality  to  full  functionality  and  from  relatively  low  cost  to  higher  cost, 
are  available  across  different  application  platforms. 

4.  A  moderate  number  (4  or  more)  of  commercial  products,  spanning  the  range  from 
minimal  functionality  to  full  functionality  and  from  relatively  low  cost  to  higher 
cost,  are  available  across  different  application  platforms. 

3.  A  moderate  number  (4  or  more)  of  commercial  products,  offering  similar  levels  of 
functionality  at  similar  cost,  are  available. 

2.  A  small  number  (2  or  3)  of  commercial  products,  offering  similar  levels  of  function¬ 
ality  at  similar  cost,  are  available. 

1 .  Only  one  commercial  product  is  available. 

Marketplace  Presence 

Marketplace  presence  refers  to  the  percentage  of  the  marketplace  covered  by  all  products 
adhering  to  the  standard,  relative  to  the  total  market  covered  by  all  products  of  all  competing 
standards. 

5.  The  standard  is  the  single  marketplace  leader.  It  has  the  highest  market  share  and 
has  no  close  competition. 
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4.  The  standard  is  in  the  marketplace-leading  group  of  standards.  It  is  one  of  a  group 
of  two  or  more  standards  that  have  similar  market  share.  No  standard  has  signifi- 
candy  higher  market  share. 

3.  The  standard  has  significant  market  share,  but  at  least  one  other  standard  has  a  more 
dominant  market  share. 

2.  The  standard  has  a  small  but  noticeable  market  share. 

1.  The  standard  has  an  insignificant  market  share  position. 

Cost  of  Standard 

Cost  of  standard  refers  to  the  cost  of  complying  with  the  standard  as  perceived  by  the  user. 

Factors  that  may  be  considered  in  evaluating  cost  include  the  inherent  cost  of  any  hardware  or 
software  technology  espoused  by  the  standard;  cost  of  adapting  the  existing  computing  envi¬ 
ronment  to  the  standard;  and  licensing  restrictions  if  any. 

5.  The  cost  of  adhering  to  the  standard  is  much  less  (25%  or  more)  than  the  cost  of 
adhering  to  other  standards  with  similar  purpose. 

4.  The  cost  of  adhering  to  the  standard  is  less  (5%  or  more)  than  the  cost  of  adhering 
to  other  standards  with  similar  purpose. 

3.  The  cost  of  adhering  to  the  standard  is  similar  to  the  cost  of  adhering  to  other  stan¬ 
dards  with  similar  purpose. 

2.  The  cost  of  adhering  to  the  standard  is  greater  than  (5%  -  24%)  the  cost  of  adhering 
to  other  standards  with  similar  purpose. 

1.  The  cost  of  adhering  to  the  standard  is  much  greater  than  (25%  or  more)  the  cost  of 
adhering  to  other  standards  with  similar  purpose. 

4.2  GUIDELINES  FOR  USING  THE  STANDARDS  EVALUATION  MODEL 

One  way  of  using  the  standards  evaluation  model  is  to  assess  standards  against  all  cri¬ 
teria  in  the  model  and  to  give  all  criteria  equal  importance  in  making  the  decision  as  to  which 
standard  should  be  chosen.  This  can  be  termed  a  “full  evaluation”.  This  section  presents  a  few 
guidelines  for  tailoring  and  streamlining  the  use  of  the  model,  so  that  the  evaluation  process 
may  be  less  strenuous  than  a  full  evaluation  would  be,  and  the  results  may  be  more  relevant  to 
a  particular  situation. 
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GUIDELINE  1:  Use  one  or  two  significant  criteria  to  narrow  the  set  of  standards  under 
consideration. 

Very  often  a  single  criterion  such  as  marketplace  acceptance  is  of  such  overriding 
importance  that  it  can  be  used  to  eliminate  candidates  that  don’t  measure  up  relative  to  it.  It  is 
possible  that  all  but  one  standard  can  be  eliminated  using  significant  criteria,  and  the  choice  will 
be  made  without  further  evaluation. 

GUIDELINE  2:  Use  only  criteria  that  are  important  in  a  given  context 

If  certain  criteria  are  not  important  in  the  context  in  which  the  standard  will  be  used, 
they  should  be  eliminated  from  the  evaluation  process.  For  example,  if  the  domain  is  mature 
and  only  established  standards  are  under  consideration,  the  quality  criteria  can  be  eliminated  in 
favor  of  the  markeq>lace  criteria.  On  the  other  hand,  if  the  technology  and  standards  in  the 
domain  are  relatively  new,  marketplace  criteria  will  not  be  able  to  discriminate  among  them, 
whereas  quality  criteria  will. 

GUIDELINE  3:  The  criteria  that  are  used  in  the  evaluation  should  be  weighted  to  reflect 
their  relative  importance. 

If  certain  aspects  of  a  standard  are  more  important  because  of  the  context  in  which  the 
standard  will  be  used,  the  standards  evaluation  criteria  can  be  weighted  to  reflect  that  impor¬ 
tance. 

The  non-functional  and  external  requirements  are  important  indicators  of  which  stan¬ 
dards  evaluation  criteria  will  be  most  important.  With  many  non-fiinctional  requirements,  the 
evaluation  criteria  most  relevant  to  achieving  them  are  obvious.  For  example,  if  cost-related 
requirements  are  important,  then  the  cost  criterion  will  be  particularly  significant. 
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For  non-functional  requirements  relating  to  portability,  scalability,  and  interoperability, 
several  standards  evaluation  criteria  may  be  significant  in  achieving  the  requirements,  as  doc¬ 
umented  in  Table  4-1. 

Table  4-1.  Evaluation  Criteria  for  Portability,  Scalability,  Interoperability 


Non-functional 
Requirement  Area 

Significant  Evaluation  Criteria 

Portability 

Completeness 

Unambiguous  Expression 

Strictly-defined  Interface 

Marketplace  Presence 

Stability 

Scalability 

Flexibility  (e.g.,  16  vs.  32-bit  format) 
Performance  (of  acceptable  products) 

Interoperability 

Completeness 

Unambiguous  Expression 

Strictly-defined  Interface 

Flexibility  (e.g.,  protocol  flexibility) 

Scope  of  Acceptance 

Marketplace  Presence 

Other  factors  to  consider  when  weighting  evaluation  criteria  include  the  scope  or 
impact  of  the  system  being  developed.  This  is  often  addressed  in  the  external  requirements.  For 
example,  if  the  new  system  is  to  have  an  enterprise-wide  impact,  it  will  probably  be  longer- 
lived  than  a  local  system  would  be.  Certain  factors  such  as  the  stability  of  the  standard,  support 
for  the  standard,  and  flexibility  of  the  standard  in  adapting  to  future  developments  become  rel¬ 
atively  more  important. 

4.3  STANDARDS  EVALUATION  MODEL  EXAMPLE 

The  following  hypothetical  situation  is  presented  to  illustrate  the  use  of  the  standards 
evaluation  model.  A  contractor  organization  has  just  been  awarded  a  contract  for  a  large 
defense  information  system,  which  is  to  incorporate  portions  of  several  legacy  systems  as  well 
as  a  significant  amount  of  new  software  development.  The  decision  has  already  been  made  to 
use  object-oriented  technology  for  the  new  development.  The  technical  and  managerial  staff 
have  experience  in  both  Ada  and  C++,  but  in  order  to  conform  to  the  DoD  mandate  have  a  pref¬ 
erence  for  using  Ada  95.  The  organization  has  decided  to  use  the  standards  evaluation  model 


to  assess  Ada  95  relative  to  the  qualities  they  believe  they  will  need  from  the  programming  lan¬ 
guage  chosen  for  the  project 

Their  first  step  is  to  identify  the  standards  evaluation  criteria  that  are  most  important  and 
relevant  to  their  situation.  The  organization  decides  to  use  the  criteria  of  completeness,  unam¬ 
biguous  expression,  strictly-defined  interface,  conformance  specification,  stability  of  standard, 
number  of  acceptable  products,  and  marketplace  presence.  This  will  support  their  goals  to  have 
software  that  is  portable  across  several  operating  systems  and  hardware  platforms,  to  have  a 
variety  of  tools  to  choose  from  that  have  been  ascertained  to  conform  to  the  language  specifi¬ 
cation,  and  to  have  stability  in  their  software  and  tool  set 

They  analyze  their  needs  and  come  up  with  the  following  desired  levels  of  assessment 
for  the  seven  evaluation  criteria  areas  as  shown  in  Table  4-2. 


Table  4-2.  Hypothetical  Desired  Levels  of  Assessment 


Com¬ 

plete¬ 

ness 

Unamb 

Express 

Strict 

Inter¬ 

face 

Con¬ 

for¬ 

mance 

Sta¬ 

bility 

#of 

Accept 

Prods 

Market 

Pres¬ 

ence 

Organizational 

Needs 

5 

4 

5 

4 

3 

4 

3 

A  technical  person  who  has  in-depth  knowledge  of  the  Ada  95  standard  is  assigned  the 
task  of  evaluating  the  standard  relative  to  the  seven  chosen  evaluation  criteria,  using  the  assess¬ 
ment  guidelines.  The  reasoning  for  several  of  the  resulting  assessment  scores  is  described 
below  and  in  Table  4-3. 

•  Strictly-defined  interface:  Most  aspects  of  the  language  are  strictly  defined  in  the 
standard.  The  few  exceptions  are  typical  of  other  programming  languages.  For 
example,  a  minimum  size  is  given  for  Integers  (16  bits)  but  no  maximum  is  speci¬ 
fied.  This  criterion  is  assessed  at  level  4.  The  Ada  95  standard  includes  several 
Annexes  which  are  optional  to  implementers,  but  this  represents  optional  functions 
or  features  rather  than  options  in  the  interface  of  a  given  feature.  Hence  it  does  not 
lower  the  assessment  level. 

•  Stability  of  Standard:  The  last  update  to  Ada  was  in  1983,  and  no  update  to  Ada  95 
is  currently  planned.  Hence  this  criterion  is  assessed  at  level  5.  There  could  be  some 
question  as  to  whether  Ada  95  is  the  same  language  (standard)  as  Ada,  but  it  is  offi¬ 
cially  considered  an  update  to  the  Ada  standard. 


•  Marketplace  Presence:  This  criterion  is  assessed  at  level  2.  Other  de  jure  and  de  fac¬ 
to  standards  such  as  C,  C-H-,  COBOL,  and  Fortran  have  signihcantly  larger  maricet 
share.  While  Ada  is  stronger  in  safety-critical,  aerospace,  and  military  weapon  sys¬ 
tem  domains,  its  overall  presence  rates  a  2. 

The  scores  for  all  seven  evaluation  criteria  are  compared  against  the  organizational 
needs  (see  Table  4-3).  Two  potential  risk  areas  are  identified,  related  to  the  criteria  of  strictly- 
defined  interface  and  marketplace  presence. 


Table  4-3.  Hypothetical  Assessment  Results 


Com¬ 

plete¬ 

ness 

Unamb 

Express 

Con¬ 

for¬ 

mance 

Sta¬ 

bility 

#of 

Accept 

Prods 

Market 

Pres¬ 

ence 

Organizational 

Needs 

5 

4 

5 

4 

3 

4 

3 

Ada  95 

5 

5 

4 

5 

5 

4 

2 

Potential  Risk 

* 

* 

The  organization  at  this  point  must  assess  the  potential  risk  attributable  to  the  lower- 
than-desired  assessment  scores.  As  a  result  of  that  assessment,  they  might  decide  to  finalize 
their  decision  to  use  Ada  95.  At  that  point,  they  could  make  use  of  the  product  evaluation  model 
to  assess  several  Ada  95  compilers  and  development  support  tools.  Alternatively,  they  might 
decide  to  use  the  standards  evaluation  model  to  assess  C++  as  a  candidate  programming  lan¬ 
guage,  look  at  its  potential  risks  to  the  organization,  and  compare  its  assessed  values  to  those 
of  Ada  95.  This  will  give  them  more  information  to  use  in  making  a  final  decision  on  which  of 
the  two  languages  will  best  suit  their  needs. 


5.  SUMMARY 


The  standards  evaluation  model  and  the  standards-based  COTS  products  evaluation 
model  have  identical  forms.  Each  provides  a  set  of  evaluation  criteria  grouped  in  broad  catego¬ 
ries.  In  addition,  the  standards  evaluation  model  includes  guidelines  for  assessing  a  standard 
relative  to  each  of  the  evaluation  criteria  in  the  model.  The  models  can  serve  as  checklists, 
ensuring  that  important  considerations  are  not  overlooked  when  selecting  a  standard  or  prod¬ 
uct.  The  use  of  the  models  can  help  provide  consistency  among  standard  and  product  evalua¬ 
tions. 

The  models  were  developed  to  help  users  select  from  among  competing  standards  or 
products.  They  can  also  be  used  to  evaluate  a  single  standard  or  product  against  organi2ational 
requirements  for  the  standard  or  product. 

The  models  are  flexible  and  can  be  tailored  to  a  given  situation.  Certain  criteria  can  be 
emphasized  or  de-emphasized,  according  to  situational  needs.  Further,  the  models  themselves 
can  easily  be  extended  or  modified  based  on  experience  in  their  use.  Criteria  can  be  added  or 
deleted,  and  assessment  guidelines  further  developed.  As  the  models  are  more  widely  used, 
they  can  evolve  to  become  more  precise  instraments  for  evaluating  standards  and  products. 
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